Mobile robot system and method for generating map data using straight lines extracted from visual images

ABSTRACT

A mobile robot is configured to navigate on a sidewalk and deliver a delivery to a predetermined location. The robot has a body and an enclosed space within the body for storing the delivery during transit. At least two cameras are mounted on the robot body and are adapted to take visual images of an operating area. A processing component is adapted to extract straight lines from the visual images taken by the cameras and generate map data based at least partially on the images. A communication component is adapted to send and receive image and/or map data. A mapping system includes at least two such mobile robots, with the communication component of each robot adapted to send and receive image data and/or map data to the other robot. A method involves operating such a mobile robot in an area of interest in which deliveries are to be made.

RELATED APPLICATIONS

This is a Continuation of U.S. patent application Ser. No. 16/908,376filed Jun. 22, 2020, now U.S. Pat. No. 11,048,267, which is aContinuation of U.S. patent application Ser. No. 15/968,802, filed May2, 2018, now U.S. Pat. No. 10,732,641, which is a Bypass Continuation ofInternational Application No. PCT/EP2016/076466, filed Nov. 2, 2016, andpublished as WO 2017/076928A1, which claims priority to European PatentApplication No. EP 15192649.0, filed Nov. 2, 2015. The contents of theaforementioned applications are incorporated by reference in theirentirety.

FIELD

The invention relates to mapping performed by a moving robot. Moreparticularly, the invention relates to mapping using multiple camerastaking visual images.

INTRODUCTION

Mapping done by mobile robots and localization of mobile robots has beenan active area of research since progress in robotics allowed forsemi-autonomous and autonomous robots. A moving robot must have a methodof at least estimating its position in space to continue motion with apurpose. The modern framework of mapping localisation comprises acoordinate system with respect to which the localization is done and apose, which is a combination of position and orientation. Localizationcan be done with respect to an absolute coordinate system (such as GPScoordinates) or a relative coordinate system (such as localization withrespect to some known location and/or object). The coordinate system canbe chosen arbitrarily, as long as it is consistent and can be convertedto some standard coordinate system (such as WGS84) if needed. Multiplesensor readings can contribute to pose calculation—it can be determinedusing GPS receivers, Lidar (light radar) sensors, cameras, odometers,gyroscopes, accelerometers, magnetometers, time of flight cameras andradar sensors. There is an important distinction to be made in thecontext of localization: it can be done based on an existing map, or itcan be done simultaneously with mapping. The latter is called SLAM(Simultaneous Localization and Mapping) and is the preferred approachwhen localization is performed while exploring previously unknownsurroundings. If a map is already available, the task becomes easier.For example, localization in indoor environments (such as an apartmentor a hospital) or structured outdoor environments (such as a factorycomplex) is easier, since a detailed map is readily available, and it isunlikely to change significantly in the short term. Localizationoutdoors, in unstructured environments (such as cities, suburbs and/orvillages) is a greater challenge. First, publicly available maps are notprecise enough for autonomous motion by the robot. Second, even when themaps exist, they are likely to get outdated very fast, as new obstaclesappear and old pathways disappear. Localization outdoors can be donewith the help of a positioning system such as GPS (Global PositioningSystem), GLONASS (Global Navigation Satellite System) or Galileo.However, the precision of these systems (available for public use) is onthe order of 1-10 meters. This would not be enough for localization usedfor autonomous robotic navigation.

Existing solutions to the localization problem strongly depend on theintended application of the object to be localized.

U.S. Pat. No. 8,717,545 B2 discloses a system using range and Dopplervelocity measurements from a lidar system and images from a video systemto estimate a six degree-of-freedom trajectory of a target. Once themotion aspects of the target are estimated, a three-dimensional image ofthe target may be generated.

U.S. Pat. No. 7,015,831 B2 discloses a method and apparatus that use avisual sensor and dead reckoning sensors to process SimultaneousLocalization and Mapping (SLAM). Advantageously, such visual techniquescan be used to autonomously generate and update a map. Unlike with laserrangefinders, the visual techniques are economically practical in a widerange of applications and can be used in relatively dynamicenvironments, such as environments in which people move.

The disclosed method can be implemented in an automatic vacuum cleaneroperating in an indoor environment.

U.S. Pat. No. 8,305,430 B2 discloses a visual odometry system and methodfor a fixed or known calibration of an arbitrary number of cameras inmonocular configuration. Images collected from each of the cameras inthis distributed aperture system have negligible or absolutely nooverlap. The system uses multiple cameras, but generates pose hypothesisin each camera separately before comparing and refining them.

Schindler, et al. (in 3D Data Processing, Visualization, andTransmission, Third International Symposium on, pp. 846-853. IEEE, 2006)discusses a novel method for recovering the 3D-line structure of a scenefrom multiple widely separated views. In this approach, 2D-lines areautomatically detected in images with the assistance of an EM-basedvanishing point estimation method which assumes the existence of edgesalong mutually orthogonal vanishing directions. 3D reconstructionresults for urban scenes based on manually established featurecorrespondences across images are presented.

Murillo, A. C., et al. (Robotics and Autonomous Systems 55, no. 5(2007): 372-382) proposes a new vision-based method for global robotlocalization using an omnidirectional camera. Topological and metriclocalization information are combined in an efficient, hierarchicalprocess, with each step being more complex and accurate than theprevious one but evaluating fewer images.

SUMMARY

The present invention is specified in the claims as well as in the belowdescription. Preferred embodiments are particularly specified in thedependent claims and the description of various embodiments.

In a first aspect, a mapping method making use of the invention isprovided. The mapping method can comprise operating at least one mobilerobot comprising at least two cameras and at least one processingcomponent. The mapping method can further comprise taking visual imageswith at least two cameras. The mapping method can also compriseextracting at least straight lines from the visual images with at leastone processing component. The mapping method can further comprisegenerating map data using the extracted features from the visual images.

In a second aspect, the invention discloses a mobile robot. The mobilerobot can comprise at least two cameras adapted to take visual images ofan operating area. The mobile robot can also comprise at least oneprocessing component adapted to at least extract straight lines from thevisual images taken by the at least two cameras and generate map databased at least partially on the images. The mobile robot can furthercomprise a communication component adapted to at least send and receivedata, particularly image and/or map data.

In a third aspect, the invention discloses an assembly of mobile robotscomprising at least two mobile robots. Each of the mobile robots cancomprise at least two cameras adapted to take visual images of anoperating area. Each of the mobile robots can further comprise at leastone processing component adapted to at least extract straight lines fromthe visual images taken by the at least two cameras and generate mapdata based at least partially on the images. Each of the mobile robotscan also comprise at least one communication component adapted to atleast send and receive data at least between the robots, particularlyimage and/or map data.

In some embodiments, the mapping method further comprises combining thevisual images and/or the extracted straight lines in a single referenceframe based on the relative placement of the two cameras. That is, thevisual images from two of more cameras can be combined into one image.This step can be done before extracting the straight lines or afterextracting the straight lines. In the latter case, only the extractedstraight lines can be combined in a single reference frame. Thereference frame can be chosen, for example, with respect to one of thecameras. The relative placement of the cameras can be computed preciselyprior to starting the mapping process. This can be useful to obtain afuller picture or map of the robot's surroundings. It can also be usefulfor determining whether any straight lines extracted from the images ofdifferent cameras belong to the same object on the images taken by therobot, that is, belong to the same object in the world.

In some preferred embodiments, the method can further compriseextracting location-related data from at least one further sensor. Saidsensor can comprise, for example at least one or a combination of atleast one GPS sensor, at least one dead-reckoning sensor, at least oneaccelerometer, at least one gyroscope, at least one time of flightcamera, at least one Lidar sensor, at least one odometer, at least onemagnetometer and/or at least one altitude sensor. The method can thenfurther comprise combining this location-related data with the data fromthe visual images to build map data. That is, in some embodiments, themobile robot can use data from its other sensors to obtain anapproximate map of an operating area it finds itself in, and then refinethis approximate map using the extracted straight lines. Alternativelyor additionally, the robot can build map data using the extractedstraight lines and other sensor readings simultaneously. This isadvantageous, as the other sensors' data can serve as a “sanity check”,or as a means to make the camera images-based map built from thestraight lines more precise and reliable. This is described in furtherdetail below.

Note that, the term “operating area” can refer to a region where therobot is moving. This can, for example be a certain street, or a patchof a street. For example, operating area can comprise a 1 to 1000 mregion that the robot is moving through. One robot run can comprise oneor a plurality of operating areas. When generating map data, the robotcan, for example, generate it based on an area with a radius of 1 to1000 m, more preferably 10 to 500 m. Once a map of one patch or area oroperating area is generated, the robot can generate a map of asubsequent patch or area or operating area.

Note that the term “camera image” can also refer to camera frames.

The mapping method can further comprise using an iterative algorithm toat least associate the extracted straight lines with physical objects.That is, when generating map data, the algorithm can identify the 2Dstraight lines from camera images with 3D physical objects or landmarkspresent in the real world. For example, the algorithm can detect that acertain number of lines all belong to the same physical object orlandmark and merge them into one. Conversely, the algorithm can detectthat a certain line only appears in one camera image and is likely tobelong to a transitory physical object such as a car, or to be caused bylight and/or camera effects. In this case, the algorithm can reject thisline and not assign it to a 3D physical object or landmark. Physicalobjects or landmarks can comprise non-transitory objects such asbuildings, fences, roads, poles, windows, doors, mailboxes, streetlights, street signs, and other objects comprising substantiallystraight features.

In some embodiments, the iterative algorithm can comprise anoptimization routine that optimizes at least one robot pose and theposition of the physical objects based on the extracted features. Thatis, the algorithm can for example take one or a plurality of robot posesand the number and positions of physical objects or landmarks as unknownvariables, and optimize their values based on the straight linesextracted from camera images. In some embodiments, the algorithm canalso take data from other sensors into consideration, for example GPSsensor or other sensors listed above and below.

Note, that in the present document, the terms “physical object” and“landmark” can be used interchangeably.

In some embodiments, the iterative algorithm can comprise at leastcombining lines belonging to the same physical object from images takenby different cameras. It can also comprise discarding lines belonging totransient objects and/or light or camera effects. That is, one landmarkor physical object can be detected by different cameras at the same timeframe and/or by the same camera at several time frames and/or bydifferent cameras at several time frames. The algorithm can be adaptedto combine the lines extracted from several images and associate them toa certain landmark with a certain fixed position on a map (with acorresponding error). Conversely, if a certain line is detected in toofew frames—be they from the same camera at different time points and/orfrom different cameras at the same or different time points, this linecan be an artefact of the light or of the camera. It could also be atransient feature such as a car (which should not comprise part of amap). In this case, the algorithm can be adapted to discard or cull thecorresponding line.

In some embodiments, each of the robot cameras can be adapted to capture3 to 5 images or frames per second, preferably 4 images or frames persecond.

In some embodiments, the robot comprises at least 4 cameras. In somepreferred embodiments, the robot comprises 8 or 9 cameras.

The map data that the robot can generate can comprise a plurality ofvectors, point features and/or grid features defined with respect to acoordinate system. This coordinate system can be tied to the robot, forexample with one of the cameras serving as the origin and the directionof motion as the x-axis. The coordinate system can also be independentof the robot, such as a GPS system for example. The grid features cancomprise extended objects that are not lines, such as, for example,geometrical shapes. For instance, in some embodiments, the street polescan be modelled as geometrical objects with a radius varying withheight. These can then be considered grid features. Furthermore,rectangular or substantially rectangular objects can be considered gridfeatures. These can comprise, for example doors, windows and/or otherobjects.

Additionally or alternatively, substantially circular, oval, triangularand differently shaped objects can comprise grid features. The map datacan simply comprise a collection of numbers corresponding to thepositions of straight lines and/or of physical objects (landmarks)and/or of robot poses with respect to some coordinate system. There canalso be a plurality of coordinate systems defined for map data. The mapdata can further comprise lines obtained on the basis of camera imagelines. That is, the lines extracted from camera images can be runthrough an algorithm to verify whether they belong to actual landmarks(that is, physical objects), and whether some of them can be combined.At this stage, the position of the lines with respect to each otherand/or with respect to the robot can be adjusted, as lines extractedfrom images taken by different cameras and/or taken at different timepoints can be combined if it is determined that they belong to the samelandmarks. Some lines can also be discarded if it is determined thatthey likely belong to transient objects (such as trucks or cars), lighteffects and/or camera effects. Map data can then comprise thiscollection of lines based on the lines extracted from images, butprocessed by a certain algorithm to comprise mostly or solely linesbelonging to landmarks, combined lines, and/or lines with an adjustedposition. Map data can further comprise error estimates associated withvectors, point features, and/or lines associated with landmarks. Mapdata can further comprise visibility information. That is, informationabout from which locations given landmarks can be seen. In other words,if, for example, a wall would obstruct a landmark or physical object, sothat it cannot be seen from locations behind the wall, this informationcan also be comprised in map data. This is practical, as when, forexample, the robot uses the map to navigate, it will not detectlandmarks of physical objects that are nearby, but obstructed. Thevisibility information can also take into account the height of therobot's cameras to determine which object can be visible at whichlocations.

In some embodiments, the communication component can be adapted toexchange map data and/or image data with a server, particularly for atleast refining map data by the server. “Refining” can refer, forexample, to actually building a map. In some embodiments, the algorithmfor building a map can be run on the server. In such embodiments, therobot can send map data comprising the extracted straight lines from thecamera images, potentially combined and arranged with respect to asingle coordinate system, to the server. The server can then run theoptimization algorithm that can compute the robot poses and thelocations of physical objects or landmarks, that is, build a map. Inother embodiments, the algorithm can run on the robot, that is, on theprocessing component of the robot, and the map data sent to the servercomprises a map of some operating area. In such embodiments, the servercan serve to combine the map of multiple operating areas into a singleneighbourhood map, town map, city map, world map, or just a map of somelarger region. “Refining” can also comprise improving the accuracy ofthe map data sent by the robot, that is running the optimizationalgorithm again, or running another algorithm to obtain better map data.Note, that the term “map data” is used herein to refer to both map datagenerated by the robot's processing component and the server. A skilledperson will understand, that generating map data can be done on therobot, on the server or on both. Intermediate map data, such as, forexample, straight lines extracted from images, can comprise map data.Map data comprising association of lines with landmarks or physicalobjects with the corresponding errors can comprise further map dataand/or final map data that can be generated either by the robot and/orby the server.

In some embodiments, the robot can be further adapted to navigate usingthe map data from the processing component and/or from the server. Thatis, the robot can use the map data to localize itself and to chart acourse for itself.

In some preferred embodiments, the robot can further comprise sensorsadapted to measure further parameters for building map data. The sensorscan comprise, for example, at least one or a combination of at least oneGPS component, at least one accelerometer, at least one gyroscope, aleast one odometer, at least one magnetometer, at least one pressuresensor, at least one ultrasonic sensor, at least one time of flightcamera sensor, and/or at least one Lidar sensor.

In some preferred embodiments, the robot can be autonomous and/orsemi-autonomous. That is, the robot can be adapted to navigate and/or todrive autonomously for a significant proportion of a given run. In someembodiments, the robot is adapted to navigate and/or drive autonomouslyfor at least 50% of its total operating time. In some other embodiments,when gathering image data for building map data, the robot can beremotely controlled by a remote operator. That is, the robot can becontrolled and/or driven by a person and/or by a server and/or by asecond processing component.

The robot can further be adapted to travel or drive with a speed of nomore than 10 km/h, more preferably no more than 8 km/h, or no more than6 lm/h, preferably between 3 and 6 km/h, or, more preferably, between 4and 5 km/h.

In embodiments comprising an assembly of mobile robots, that is two ormore mobile robots, the robots can be adapted to communicate, orexchange data via their respective communication components. Thecommunication can be direct and/or routed through a server. In suchembodiments, the data sent between the robots can be combined into a mapof multiple operating areas. As mentioned above, this can, for example,mean that the server can receive map data from at least two mobilerobots and combine this data into a map or map data of both of theiroperating areas. In some other embodiments, a first robot can collectimages of an operating area, extract straight lines, build map data outof them, and then transmit said data to a second robot so that it cannavigate the same operating area using the map data of the first robot.The transmission of map data can be done directly between the robotsand/or done via a server. That is, in some embodiments, the assembly canfurther comprise a server adapted to at least communicate with therobots via their communication component, particularly to receive mapdata and/or image data from a first robot, process it and send processeddata to a second robot when requested. In some embodiments, “processingdata” by the server refers simply to assigning it to the respectiveoperating area in relation to other operating areas. In otherembodiments, “processing” can refer to creating further map data and/orimproving the obtained map data.

Below follows a detailed description of one preferred embodiment of themapping method that can be implemented at least partially by the mobilerobot.

The mapping method can be implemented as a probabilistic algorithm thattakes certain input variables and returns certain output variables.Input variables can comprise, for example, the robot's pose, the linesdetected on the visual images and/or certain objects with known realworld coordinates. The output variables can comprise, for example,location of landmarks on a robot-dependent coordinate system, i.e. a mapof a certain region. Landmarks can comprise structures such asbuildings, doors, fences, poles, street signs, windows and otherpreferably man-made structures easily identifiable by straight lines.Such landmarks can generally be separated into two types: line typelandmarks and pole type landmarks. That is, landmarks that aresubstantially vertical can comprise pole type landmarks and landmarksthat are substantially horizontal can comprise line type landmarks.

Suppose a robot is driven through some area. This can be referred to asa robot run. Along the way it can pick up sensor readings such as theGPS position, movement distance from the odometer, rotation informationfrom the gyroscope, camera images, accelerometer data, and potentiallyother readings. Two quantities can be defined in this context: the map(M) and sensor readings (R). The map M is a collection of all thelandmarks the robot can see with its camera. The map can also containthe exact path the robot took. It can be simply defined as a vector ofnumbers:M=(m ₁ , . . . ,m _(|M|))

The values of M are unknown in the mapping problem, so it is thevariable to be determined. Furthermore, it is not known how manylandmarks there are or the type of those landmarks. The sensor readingsR can likewise be defined as a vector of numbers,R=(y ₁ , . . . ,y _(|R|))

It is the result of an actual robot run and is a constant quantity inthe following Bayesian statement:

${P\left( {M❘R} \right)} = \frac{{P\left( {R❘M} \right)}{P(M)}}{P(R)}$

The statement is simply the Bayes theorem that establishes therelationship between conditional probabilities. It establishes therelationship between conditional probabilities P(A|B) (probability of Agiven B) and P(B|A) (probability of B given A). This is well known tothe person skilled in the art.

That is, if some map was known, the relationship of its probabilitygiven the sensor readings to the probability of having such readings ormap independently at all could be established. These probabilities canbe almost impossible to numerically quantify. However, their exactvalues are not actually needed. Only the “best” map should be found,defined as:

$M_{BEST} = {\arg\max\limits_{M \in {\mathbb{M}}}{P\left( {M❘R} \right)}}$

That is, the map maximizing the probability of the map M given theobtained sensor readings R. From a camera image, a series of lines oredges is obtained. These lines correspond to the landmarks on the map.However, it is not known which lines are associated to which landmarks.Some of the lines are also noise. The problem set can thus be summarizedas follows:

-   -   determine the number of landmarks |M| and their types    -   determine which 2D lines from camera images are associated to        which landmarks    -   find the best map M_(BEST) given the sensor readings.

The map M can contain two parts: the path the robot took and thelocation of the landmarks, so thatM=(M _(P) ,M _(L))

It can be natural to quantize the path into a set of poses coincidingwith the times the camera images were taken. So, if N_(P) camera imageswere taken (at times t₁, . . . , t_(N) _(P) ), then the robot path partof the map can look asM _(P)=(x ₁ ^(P) ,y ₁ ^(P) ,z ₁ ^(P),α₁ ^(P),β₁ ^(P),γ₁ ^(P) , . . . ,x_(N) _(P) ^(P) ,y _(N) _(P) ^(P) ,z _(N) _(P) ^(P),α_(N) _(P) ^(P),β_(N)_(P) ^(P),γ_(N) _(P) ^(P))where a pose is represented by six values x_(i) ^(P), y_(i) ^(P), z_(i)^(P), α_(i) ^(P), β_(i) ^(P), γ_(i) ^(P) (the robot position and theEuler angles).

As mentioned above, it can be useful to categorize the landmarks intoline type and pole type. Line type landmarks can be formed by edges(building corners, pavement edges, window edges, etc). In 3D space theycan be represented by four numbers: position x_(i) ^(L), y_(i) ^(L) andorientation Euler angles α_(i) ^(L) and β_(i) ^(L), provided a constantreference frame (describing the line's “vertical space”) represented byrotation matrix R_(i) ^(L)∈

^(3×3) and origin c_(i) ^(L)∈

^(3×1) are given.

If there are N_(L) line type landmarks, then the landmark part of themap vector isM _(L)=(x ₁ ^(L) ,y ₁ ^(L),α₁ ^(L),β₁ ^(L) , . . . ,x _(N) _(L) ^(L) ,y_(N) _(L) ^(L),α_(N) _(L) ^(L),β_(N) _(L) ^(L))To conclude, if there are N_(P) poses and N_(L) landmarks, then the linetype landmark part of the map M is simply a vector of 6·N_(P)+4·N_(L)numbers (assuming that the origin and reference frame for the line typelandmarks are given).

Vertical or substantially vertical lines can comprise pole typelandmarks. Many vertical lines can be generated by poles—lamp poles,traffic sign posts, fence posts, trash cans, bollards, etc. Theselandmarks can also be modelled and added as input variables into the mapM.

Note that associating lines detected from camera images to landmarks canbe challenging. Some lines belong to the same landmark, while otherlines don't belong to landmarks at all (they can be detected due to ashadow on the ground, a moving car, a light trick, or something else).To accurately associate lines to actual landmarks, lines can be trackedacross multiple robot poses (i.e. as the robot moves and the camerastake images at different positions and orientations of the robot), somelines can be merged together, some lines can be discarded, color can becompared on each side of the lines. The association can be done as aniterative algorithm smoothly adjusting landmark candidates and cullingthem as needed.

The sensor readings obtained during a robot run are constant. Therefore,in the original Bayesian statement, P(R)=const. The quantity P(M)corresponds to the probability of getting a certain map M from the setof all possible maps. This set is unrestricted, and therefore verylarge. In practice, only maps that are physically possible within thecontext of the chosen data model are considered. The probability ofobtaining such M from the set of all maps is about the same. Therefore,also P(M)≈const. What is left to estimate isP(M|R)∝P(R|M)

That is, the probability of the map given certain readings isproportional to the probability of the readings given the map. P(R|M)can be quantified in the following way. P(R|M) is the probability ofgetting sensor readings R if given map M. Sensor readings containerrors. A further approximation can be made by stating that the sensorreadings are approximately normally distributed, that isR˜N(μ,Σ)where μ are the error free values (ground truth that is not known) and Σis the covariance matrix that can be derived from the properties of thesensors and the readings taken. While μ is not known, given the bestmap, its approximation can be calculated, that is μ≈μ(M_(BEST)).

The form of the probability P(R|M) can now be approximately stated as:P(R|M)≅C _(w) e ^(−½(μ(M)-R)) ^(T) ^(Σ) ⁻¹ ^((μ(M)-R)),where C_(w) contains the normalization constant. The goal is to maximizeP(M|R)—this is equivalent to maximizing the value P(R|M), or,equivalently, minimizing its negative logarithm:

${{\ln{P\left( {R{❘M}} \right)}} = {{{- \frac{1}{2}}\left( {{\mu(M)} - R} \right)^{T}{\sum\limits^{- 1}\left( {{\mu(M)} - R} \right)}} + C_{L}}},$${M_{BEST} = {\underset{M \in {\mathbb{M}}}{\arg\min}\left( {{\mu(M)} - R} \right)^{T}{\sum\limits^{- 1}\left( {{\mu(M)} - R} \right)}}},$${{S(M)} = {\left( {{\mu(M)} - R} \right)^{T}{\sum\limits^{- 1}\left( {{\mu(M)} - R} \right)}}},$So, to find the best map M_(BEST), the sum S(M) should be minimized. Theproblem of finding the best map has been reduced to one of nonlinearminimization. The process can run as an iterative optimization algorithmaiming to maximize the overall probability of obtaining a certain mapgiven the known sensor readings.

In some preferred embodiments, the method comprises an error cappingstep, wherein the errors associated with the robot's sensors are capped.Preferably, the errors associated with systematic sensor errors and/orinvalid physical object (or landmark) associations of the camera imagelines are capped.

Looking at different maps M and determining which one minimizes the sumS(M) should yield the most likely map, given that the sensor readingsare normally distributed. That is, that the errors associated with thevalues that the sensors obtained are normally distributed. However, theerrors associated with sensor readings can be inherently non-Gaussian,or may contain various systematic errors due to external and internalcauses. For example, systematic errors can include the following: aconstant or slowly-decaying offset; bias of the sensor in a certaindirection; non-modelled or incorrectly sensed robot behaviour (forexample, wheel slippage can result in erroneous odometry reading);incorrect feature association to a landmark (for example, a lineextracted from a camera image can be associated to a wrong landmark);sensor reading timing offset (a sensor reading can actually be from atime before or after the sensor-reported time); incorrect calibration ofa sensor; change of sensor orientation within the robot reference framedue to the robot flexing during movement or due to vibration; sensormisbehaving due to external environmental factors (for example,magnetometer readings can be unreliable when the robot passes close by alarge metal object or an electrical conduit); sensor misbehaving due toinherent problems or defects in its design or manufacture etc.

These sensor reading errors can result in outliers that can severelydistort the sum S(M), resulting in an incorrect map, or even in acomplete failure of the optimization algorithms. It is assumed that mostsensors produce correct results (normally distributed with knowncovariance) most of the time. However, it is not known which readingsare normal and which contain systematic errors; therefore, to avoidfailure of the algorithm or producing the wrong maps, a technique forcapping the sensor errors can be used.

The sum S(M) can be diagonalized (as is known in the art and obvious forthe skilled person) into

${S(M)} = {\sum\limits_{i = 1}^{❘R❘}\left\lbrack \left( \frac{g_{i}\left( {M,R} \right)}{\sigma_{i}} \right) \right\rbrack^{2}}$

To compensate for the systematic errors, this sum of squared scalederrors can be modified by introducing an error capping function α(x), sothat

${S(M)} = {\sum\limits_{i = 1}^{❘R❘}\left\lbrack {\alpha\left( \frac{g_{i}\left( {M,R} \right)}{\sigma_{i}} \right)} \right\rbrack^{2}}$

The function α(x) can be chosen so that α(x)∝x when |x|<x_(L), wherex_(L) is in the range of linearity (so that the function isapproximately linear around the origin), and the function is constrainedso that |α(x)|<ϵ when |x|<x_(C), where x_(C) is the maximum possiblesensor error and ϵ is the maximum allowed error (that can be adjustedaccording to testing for example). One function that can be used here isthe arctangent. Similarly behaving functions can also be used. Inpreferred embodiments, the function α(x) can be strictly monotonic. Inother preferred embodiments, the function can have a continuous firstderivative. The function α(x) can be scaled to pick a suitable range oflinearity x_(L) and the maximum error value ϵ. Different parameters oflinearity and of maximum error can be used for different sensors. Theparameters may also be changed dynamically when additional informationis known about the operation of the sensor of during the minimizationalgorithms as the map M converges to its best value. In preferredembodiments, the linear range x_(L) can be within 1-10 and the maximumerror value E can be within the range 10-100 (assuming sensor errors arescaled to realistic values). Aside from the diagonal form of S(M) (asshown above), the error capping can instead be applied directly to theelements of vector μ(M)—R if suitable scaling is used. Then, the errorcapping can be applied without transforming S(M) to its diagonal form.

This modification allows for large occasional systematic errors in thesensor readings and feature associations to landmarks, while at the sametime yielding to the standard algorithms used for nonlinearminimization. Such algorithms can comprise, for example, the gradientdescent method, nonlinear conjugate gradient methods, quasi-Newtonmethods, the Gauss-Newton method, the Levenberg-Marquardt method etc.The modification works with Newton-based methods, because the cappingfunction is linear around the origin. The modification can work bestwith gradient-based methods when the capping function α(x) is strictlymonotonic. The error capping step of the algorithm can be particularlyadvantageous, as it can increase the robustness of the map finding.Using error capping allows the algorithm to cope with systematic sensorand association errors while still producing accurate maps. It furtherallows for use of standard and efficient optimization algorithms such asthe Levenberg-Marquardt for example. After several iterations of theoptimization algorithm have been run, and the resulting map M is veryclose to M_(BEST), the sensor readings with large systematic errorsand/or invalid landmark associations can be easy to differentiate fromthe rest of the data. These sensor readings and/or invalid landmarkassociations can then be removed from the sum S(M). After this, theoptimization algorithm can be run again to further improve the mapquality.

In some other preferred embodiments, the error capping can comprise astandalone method for modifying error terms in least squares-basedoptimization algorithms. The method comprises acquiring a plurality ofdata points with corresponding error estimates by measuring a physicalsystem. The method further comprises constructing a model comprising aplurality of parameters to be determined, said model describing thephysical system. The method further comprises computing residual errorsof the data points using the model and its parameters. The methodfurther comprises scaling the residual errors by data point errorestimates. The method further comprises applying an error cap functionto the scaled residual errors. The method further comprises summing thesquared capped scaled residual errors. The method further comprisesminimizing the sum by changing the model parameters.

The model parameters can comprise, for example, the map in the casewhere the error capping is used within the mapping method as describedin the present document. That is, the landmarks and/or poses of therobot on a certain coordinate system.

The model can comprise an approximation of the physical system. That is,the model can provide a framework of relations between the measurablequantities. However, the precise mathematical details of the model canbe unknown to start with. These can comprise coefficients, exponents,angles, lengths and so on. Those unknowns can comprise the parameters(in the current application, the map).

In some embodiments, the error cap function is strictly monotonic. Theerror cap function can comprise a continuous first derivative. The errorcap function can be approximately linear around the origin. The errorcap function can be bounded from below and from above by a maximumallowed error value. The error cap function can comprise an arctangentfunction. The error cap function can also be as described above and/orbelow in the figure description.

Though in the present document, the error capping method is described asa part of the mapping method, a skilled person will understand that itcan be applied more generally to a wide array of optimizationalgorithms. It is particularly advantageous, as it can allow usingstandard algorithms such as the Levenberg-Marquardt ones for complexproblems (such as the present mapping problem). Without the errorcapping method, the standard algorithm can fail or return sub-optimalresults. However, it can be much more robust with this method.

In some embodiments, the method can be used to cap systematic errorscomprised in sensor readings. In such embodiments, it can be used toimprove the robustness of an optimization algorithm and to increase thelikelihood of obtaining parameters that conform the closest to thephysical system.

In some embodiments, the method can be used in conjunction with theLevenberg-Marquardt algorithm to at least compute a non-linear leastsquares problem.

In some other embodiments, the method can be used in conjunction with agradient-based optimization algorithm to at least compute a non-linearleast squares problem.

In some preferred embodiments, the method can be used as part of theoptimization algorithm to find the best possible map of a mobile robot'ssurroundings. That is, this use of the error capping method is describedin more detail above as pertaining specifically to the current mappingmethod.

The method can be used as part of the localization algorithm to find amobile robot's position and orientation with respect to a map. Forexample, the error capping can be used when localizing the robot usingthe map built according to the presently described method.

The method can be used as part of the optimization algorithm forcalibrating the positions and orientations of a plurality of camerasfixed with respect to each other. That is, a mobile robot with aplurality of cameras can require a calibration of said cameras. This canbe advantageous, as knowing the precise camera positions with respect toeach other can yield more accurate combinations of images taken bydifferent cameras for example.

In some embodiments, data points can comprise at least one or acombination of sensor readings such as GPS coordinates, altitude,odometry data, and/or straight lines extracted from camera images.

In some embodiments, parameters can comprise a map comprising aplurality of landmarks placed on a coordinate system.

In some embodiments, the least squares-based optimization algorithm canbe used to generate map data of a mobile robot's surroundings using atleast sensor data from the mobile robot and wherein parameters comprisea map comprising the path taken by the mobile robot. In someembodiments, the errors capped comprise at least one or a combination ofsystematic errors associated with sensors of a mobile robot such as aGPS component and/or errors associated with associating lines extractedfrom visual images to landmarks on a coordinate system.

In some embodiments, the method can comprise diagonalizing a covariancematrix associated with the error estimates prior to scaling the residualerrors by data point error estimates. That is, in some more complexleast-squares problems, the errors associated with data points cancomprise a covariance matrix. In such embodiments, the error capping canbe performed after diagonalizing this matrix. Then, the error cappingfunction can be applied to the diagonal elements and the method canproceed as before.

In some embodiments, the method can be used to cap association errorsarising from associating data points with incorrect model components.That is, in the present mapping method application, this can refer toerrors arising from false associations of lines to landmarks. In suchembodiments, the model can comprise a map and model components compriselandmarks and/or physical objects and/or path poses.

The mapping algorithm can be run on “patches” or “regions” of the totalpath that the robot is taking. For example, the mapping algorithm can berun on 300 robot poses. If the robot travels at about 5 km/h (1.4 m/s)and there are about 4-5 poses per second (as poses can be measuredsimultaneously with the camera images being taken), 300 poses cancorrespond to a patch of about 94 meters. In general, one patch cancomprise between 100 and 10000 robot poses, but preferably around100-1000 robot poses.

In some preferred embodiments, the method can be used as a SLAM methodby an autonomous and/or semi-autonomous mobile robot. SLAM refers tosimultaneous localization and mapping. That is, in some embodiments, themethod can be run on the robot as it navigates autonomously and/orsemi-autonomously. As the algorithm can also yield the path that therobot took, the robot can localize itself and build map data of itssurroundings simultaneously. This can be particularly advantageous, asthe robot can then just navigate through an unfamiliar area and buildmap data of it without previously knowing what the area looks like. Insome embodiments, the invention provides a mobile robot comprising (i)at least one memory component comprising at least map data, (ii) atleast two cameras adapted to take visual images of an environment, and(iii) at least one processing component adapted to at least extractstraight lines from the visual images taken by the at least two camerasand compare them to the map data to at least localize the robot.

In some embodiments, the invention provides an assembly of mobile robotscomprising at least two mobile robots, and each of the robots comprising(i) at least one memory component comprising at least map data, (ii) atleast two cameras adapted to take visual images of an environment, (iii)at least one processing component adapted to at least extract featuresfrom the visual images taken by the at least one camera and compare themto the map data to at least localize the robot, (iv) and at least onecommunication component adapted to send and receive data between therobots, particularly image and/or map data.

In some embodiments, a mapping method making use of the invention isprovided. The mapping method can make use of any features of the systemlisted above and below. The mapping method comprises (i) operating atleast one mobile robot comprising at least two cameras and at least oneprocessing component, (ii) taking visual images with the at least onerobot's at least two cameras, (iii) performing preprocessing on a filegenerated by combining visual images from the at least two cameras, (iv)extracting features from the individual visual images with at least oneprocessing component, and (v) building map data using the extractedfeatures from the visual images.

In some embodiments, the invention provides a mobile robot comprising(i) at least one memory component comprising at least map data, (ii) atleast two cameras adapted to take visual images of an environment, and(iii) at least one processing component adapted to at least extractfeatures from the visual images taken by the at least two cameras andcompare them to the map data to at least localize the robot.

In some embodiments, a localization method making use of the inventionis provided. The localization method can make use of any features of thesystem described herein. The localization method comprises (i) operatingat least one mobile robot comprising at least two cameras, at least onememory component, and at least one processing component, (ii) takingvisual images with the at least one robot's at least two cameracomponents, (iii) performing preprocessing on a file generated bycombining visual images from the at least two cameras, (iv) extractingfeatures from the individual visual images with at least one processingcomponent, (v) comparing the extracted features with existing map datastored on the at least one robot's at least one memory component, and(vi) localizing the at least one robot using the comparison in (v). Insome embodiments, a localization method making use of the invention isprovided. The localization method can make use of any features of thesystem described herein. The method comprises (i) providing at least onemobile robot comprising at least one dead reckoning component, at leasttwo cameras, at least one memory component, and at least one processingcomponent; and (ii) taking visual images with the at least one robot'sat least two camera components; and (iii) extracting features from theindividual visual images with at least one processing component; and(iv) obtaining location related data from the extracted features in(iii); and (v) receiving location related data from the at least onedead reckoning component; and (vi) combining location related dataobtained from the features extracted from the visual images in (iv) andlocation related data received from the at least one dead reckoningcomponent in (v) weighted based on errors associated with each of them;and (vii) forming a hypothesis on the robot's pose based on the combineddata in (vi).

The mobile robot can be an autonomous and/or a semi-autonomous robot.The robot can be land-based. The robot can be adapted to move on theground. In particular, the robot can be wheeled and adapted to move inunstructured urban areas that include moving and stationary obstacles.In one embodiment, the robot comprises a set of at least 4 (four) wheelsmounted on a frame. The robot can further comprise a body mounted on theframe. The robot can further comprise lights mounted on the body and/oron the frame. The lights can be LED lights. The lights can assist therobot with localization and/or navigation on short scales and/or withobstacle detection. The lights can illuminate the environment in whichthe robot operates in the dark and/or dusk. Additionally oralternatively, the robot can comprise at least one microphone and/or atleast one speaker, for communicating with its surrounding, for examplepedestrians in an urban area, such as on a pedestrian path.

Cameras and other components described herein can be mounted on theframe and/or on the body of the robot. In one particular embodiment, thedimensions of the robot are width: 40 to 70 cm, such as about 55 cm,height: 40 to 70 cm, such as about 60 cm, length: 50 to 80 cm, such asabout 65 cm. The invention can comprise more than one robot. In oneembodiment, the robot can operate autonomously during most of itsoperation, such as about 95% of the time, about 97% of the time, about98% of the time, or about 99% of the time. The robot can be adapted totravel with a speed of no more than 10 km/h, or no more than 8 km/h orno more than 6 km/h. In a preferred embodiment, the robot drives with aspeed between 3 and 6 km/h and/or between 4 and 5 km/h. In a preferredembodiment, the robot can be used for delivery purposes. The robot cancomprise an enclosed space within its body where at least one deliverycan be stored during the transit. The robot can further comprise asecure access device for providing access to the space. This device canbe a lock and/or a closure mechanism controlled by a secure interface.The robot and the delivery can have a combined weight of no more than 40kg, such as no more than 35 kg, such as no more than 30 kg, such as nomore than 25 kg. In a preferred embodiment, the robot and the deliveryhave a combined weight of 10-25 kg, more preferably 10-20 kg.

The memory component of the mobile robot can comprise a Random AccessMemory (RAM) device. It can be a standalone component or, in a preferredembodiment, an integrated part of a System on a Chip (SoC). The memorycomponent preferably stores at least map data that can relate to therobot's current and/or prospective operating area. The operating area,also referred to as area of operation, can be fixed or can be modifiedduring the course of the robot's operation. In a preferred embodiment,the operating area comprises an inhabited region that can consist of acity, a village, a neighbourhood and/or an arbitrarily defined part of acity. The operating area can also comprise regions with standalone humanhabitations and/or regions frequented by humans. The map data that canbe stored within the memory component can comprise a plurality ofvectors, straight lines, point features and/or grid features definedwith respect to some coordinate system. The map data can comprise twopoints and/or a point and a direction defined with respect to anarbitrary but fixed coordinate system. The coordinate system can be aCartesian coordinate system approximating such a part of the globe sothat the curvature does not introduce significant error. The coordinatesystem can be converted to a GPS coordinate system (for example WGS84)and/or other standard systems. The map data can also thus be convertedto a standard coordinate system such as GPS coordinates.

The cameras on the robot are typically adapted to take visual images ofthe surrounding environment of the robot within its operating area.Typically, the surrounding environment comprises closely located visualobjects, such as objects that are located within a radius of about 1 kmor less, 500 m or less, 300 m or less, 200 m or less or 100 m or lessfrom the robot. Accordingly, the robot can be adapted to take andprocess images of the surroundings of the robot that are within a radiusof about 1 km or less, 500 m or less, 300 m or less, 200 m or less or100 m or less. The environment is typically an unstructured outdoorenvironment that changes with time and the geographical surroundings ofthe robots as it travels along its path. The environment can also be atleast partially indoor, or under a roof, for example if the robot istravelling through a mall, garage, apartment complex, office buildings,or the like.

The cameras of the mobile robot can be for example similar to smartphonecameras. They can be adapted to capture 1-10 images per second, morepreferably 3-5 images per second or more preferably 4 images per second.The camera viewing angles can be 10°-120°, more preferably 40°-100°,more preferably 60° by 80°. The robot can comprise a plurality ofcameras. In a preferred embodiment, the robot comprises at least 4(four) cameras. In a more preferred embodiment, the robot comprises 9(nine) cameras. The cameras can be placed anywhere around the body ofthe robot, preferably in positions optimizing the viewing angles of thedifferent cameras. Some cameras can be stereo cameras. In a preferredembodiment, one pair of front cameras are stereo cameras. In a morepreferred embodiment, the robot comprises 4 (four) pairs of stereocameras. In this preferred embodiment, the stereo cameras are positionedin the front of the robot, on both sides of the robot and on the back ofthe robot. One more camera is positioned in the front of the robot. Thestereo cameras can be used to triangulate objects captured in the visualimages. Depth perception of the visual images can be improved withstereo cameras. The separation between the stereo cameras can be 5-20cm. In a preferred embodiment, the separation between the front and backstereo cameras is 5-10 cm and the separation between the sides stereocameras is 15-20 cm. The cameras can be placed on the robot so as totake landscape orientation visual images and/or portrait orientationvisual images. Landscape orientation visual images can be understood tomean visual images wherein the wider camera capture angle isapproximately parallel to the ground and the narrower camera captureangle is approximately perpendicular to the ground. In a preferredembodiment, the side cameras are placed in a portrait orientation andthe front and back cameras are placed in a landscape orientation. In oneembodiment, the cameras can take visual images of an unstructuredoutdoor environment. This environment can comprise at least one or anycombination of pedestrian paths comprising stationary and movingobstacles (for example pedestrians, animals, strollers, wheelchairs,mailboxes, trash cans, street lights and the like), vehicle roadsincluding vehicles, bicycles, traffic lights and the like and/or othertraversable environments such as parking lots, lawns and/or fields.

The use of a plurality of cameras to map the surroundings of the robotcan be particularly advantageous. A plurality of cameras can be pointedin a plurality of directions, therefore covering a larger part of therobot's surroundings. Furthermore, if one camera comprises a defectand/or gets blocked, the other cameras can allow the robot to stillrecord enough data to build a map and/or to perform localization. Thiscan be particularly relevant for a robot operating outdoors, as weatherconditions such as sunlight, rain, snow, fog, hail or similar canobstruct the view of the cameras. Multiple cameras are also particularlyadvantageous for the current inventions, as some landmarks and/orphysical objects can be seen on images taken by different cameras. Thiscan be used by the algorithm to confirm that those lines do belong to alandmark and/or to a physical object. Conversely, if a line or severallines are seen only on one camera, the lines can be an artefact of thelight and/or a transient object captured only by one camera. This isvery advantageous when lines are being associated to landmarks and/orcompared with an existing map.

The processing component can be part of and/or comprise a System on aChip (SoC), for example similar to smartphone processors. The memorycomponent can be part of the same SoC. The processing component can beadapted to localize the robot using the visual images captured by thecameras. The processing component can preferably also be adapted toextract features from the visual images captured by the cameras. In onepreferred embodiment, the features can be straight lines. The straightlines can be extracted by applying an edge detection algorithm (such asCanny algorithm for example) followed by a line extractor algorithm(such as Hough transform for example) to the preprocessed visual images.Preprocessing can include combining the images, sharpening the images,smoothing the images and/or adding contrast to the images. It can alsoinclude any adjustment to image color scheme.

The extracted features can be used to build a map of the robot'soperating area and/or to localize the robot using an existing map. A mapcan comprise a collection of vectors and/or lines and/or line segmentsand/or point features and/or grid features defined with respect to somecoordinate system. The map can be preloaded onto a memory component ofthe robot. Alternatively, the map is downloaded onto the memorycomponent during operation. The map can be downloaded in fragments asthe robot is moving between geographical areas.

The cameras on the robot can take visual images of the robot'ssurroundings during its roving in an operating area. The cameras can beadapted to take images with a frequency of 1 to images per second, suchas 2 to 8 images per second, or 3 to 7 images per second, or 3 to 5images per second, or 4 to 6 images per second. In one embodiment, thecameras are adapted to take images at a frequency of 4 images persecond. Preferably, image capture is performed continuously during therobot's operation at the described frequency, i.e. the robot ispreferably adapted to take images continuously using at least one, andpreferably all, of the cameras during its operation. The visual imagescan then be combined into one file and preprocessed. Preferably, theimages from different cameras are taken simultaneously. This can meanthat the time difference of the different images to be processed isconsiderably shorter than the time between successive images that areprocessed. After preprocessing, the file containing preprocessed imagedata is separated into individual image files representing the differentcameras, and the straight lines are extracted from the individualimages. The straight lines are then combined and used to build map dataof the robot's operating area.

In yet another embodiment, the robot already has stored map data on itsmemory component. The extracted straight lines are then compared withstored map data and run through a particle filter to model theprobability distribution of the robot pose. An optimal pose is thenchosen based on the comparison.

The processing component is adapted to localize the robot with an errorof at most 10 cm. In a preferred embodiment, the processing component isadapted to localize the robot with an error of at most 5 cm. In a morepreferred embodiment, the processing component is adapted to localizethe robot with an error of at most 3 cm. The precision of thelocalization can depend on the number of the cameras and/or on theknowledge of the relative positions of the cameras and/or on thecalibration of the system. Localization can be more precise for objectslocated closer than for objects located further away.

The processing component can combine the features extracted from visualimages taken by different cameras into a coherent map and/or localizeusing the combined extracted features. The processing component isadapted to provide instructions about navigation of the robot, by usingvisual localization based on the extracted features, as well as mapinformation and information about the intended destination of the robot.Navigation can include changing the pose of the robot (6degree-of-freedom position and orientation data) by moving in somedirection and/or rotating. Navigation can further include makingdecisions about the best course of movement and adapting those decisionsbased on the localization information.

The robot can further comprise a plurality of sensors adapted to measuredifferent parameters related to the environment and/or to thelocalization of the robot. The sensors can comprise at least one or anycombination of at least one GPS component, at least one accelerometer,at least one gyroscope (in a preferred embodiment 4 (four) gyroscopes),at least one odometer, at least one magnetometer, at least one time offlight camera and/or at least one Lidar sensor. A preferred embodimentcomprises at least one of all of those sensors. In a preferredembodiment, the sensors measure data related to the pose of the robot.The processing component localizes the robot by first processing thesensor data for an approximate pose estimate, and then improving thisestimate by using visual localization. The pose improvement can forexample be done by using a particle filter. The features extracted fromthe visual images can be compared to a map comprising featurescorresponding to a certain set of different poses.

The particle filter can then select the most likely pose estimate fromthe set based on the likelihood of each pose. The processing componentcan be adapted to localize the robot based on an iterative algorithmestimating robot pose at set time intervals. This iterative algorithmcan rely on an application of the particle filter method. A hypothesison a robot's pose can include data from at least one sensor such as atleast one camera, a GPS component, an odometer, an accelerometer, a timeof flight camera, and/or a magnetometer. A sensor's data can further beabsolute (such as data from a GPS component and/or data obtained fromvisual camera images for example) or relative to previous robot pose(such as data from odometers and/or gyroscopes for example).

The robot can further comprise a pressure sensor. The pressure sensorcan be used for precise altitude-based localization. In one embodiment,another pressure sensor can be located at a known location within therobot's area of operation, for example at a hub. The hub can be aphysical location (for example a parking lot), a physical structure (forexample a house, a warehouse, a shipping container, a barn, a depotand/or a garage), and/or a mobile structure (for example a truck, atrailer and/or a train wagon). The hub can serve as a storage,maintenance, repair, recharging and resupply station for the robot. Onehub can comprise one or more robots. In a preferred embodiment, one hubcan service a plurality of robots, such as 20-200 robots. With apressure sensor placed at the location of the hub, a precise altitudereference is established, and the localization of the robot can beimproved by comparing the data from the robot's pressure sensor to thedata from the hub's pressure sensor.

The robot can further comprise a communication component adapted toexchange data with one or more server, particularly image and/or mapdata. The server can comprise multiple servers and/or a cluster ofservers and/or one or more cloud servers. In one preferred embodiment,the server is a cloud server. In another embodiment, the servercomprises a cluster of servers, some of which can be cloud servers. Theserver can store, analyse and/or send out data, such as for example mapand localization related data. The server can also perform calculations,for example calculations related to the generation of a geographicalmap, localization calculations, and/or route calculations for the robot.The communication component can comprise at least one slot for aSubscriber Identity Module (SIM card), preferably two slots for two SIMcards. The use of two SIM cards is an advantage, since it increasesreliability and allows for simultaneous communication via both SIM cardsfor larger and/or faster data transmission. In a preferred embodiment,two different mobile operators are used for operation using the two SIMcards. In this case, if one mobile operator does not provide coverage insome part of the robot's area of operation, the robot can stillcommunicate via the other SIM card.

The robot can further be adapted to receive navigation instructions fromthe server at specific intervals and/or after requesting input from theserver. In one embodiment, the robot receives navigation instructionsevery 50-150 meters. The robot can further send a request for input tothe server when faced with an unfamiliar situation. The robot can alsorequest manual input about its navigation, for example when facinghazardous conditions such as crossing a street. During such manualoperation, a remote operator can provide navigation instructions to therobot and direct it through the hazard, such as across the street. Oncethe robot has reached a safe environment, the operator can instruct therobot to resume autonomous navigation. The operator can furthercommunicate with people in the immediate surroundings of the robotthrough the microphone and speakers that can be mounted on the robot.The robot can however continue to update its localization during manualcontrol.

In another embodiment, the invention discloses an assembly of mobilerobots comprising at least two mobile robots. The robots can be asdescribed above. The robots can be adapted to communicate with eachother via the communication module. The communication can be routed viathe server. The data sent between the robots and/or between the robotsand the server can be combined into a map of an operating area and/or ofmultiple operating areas. The coordinates used for each map data can bedifferent, but they can each be converted into a standard coordinatesystem and/or combined in one unified arbitrary coordinate system.Multiple operating areas can correspond for example to different partsof a city. An operating area can comprise one hub and/or multiple hubs.The robots benefit from the map data gathered by the other robots viathe map data exchange. The server can coordinate the exchange of mapdata. The server can further store map data and unify it into a globalmap. The server can send out map data to the robots based on theiroperating area. The map data can further be updated by the robots if thevisual images taken by the cameras demonstrate a consistent change inthe extracted features. For example, if new construction work is takingplace within an operating area, the map of this operating area can beupdated correspondingly.

In some preferred embodiments, the method can comprise a plurality ofoptimization algorithms. That is, the overall goal of the method can beto generate map data which can comprise landmarks with their locationdetermined by the lines extracted from camera images. Therefore, theoverall goal can be to find as many landmarks as possible with the bestpossible precision. At each step of the process, the best map can befound. This map can be the best given the step's assumptions on whichlines correspond to landmarks and which should be removed. Therefore,the best map can be found repeatedly, each time making new associationsand removing bad associations, as the map can improve with every step,provided the algorithm is functioning correctly. In this way, findingthe best map can be part of an iterative algorithm which repeats at eachstep of an overlying iterative algorithm that has as a goal themaximisation of the landmarks located on the map.

In some embodiments, the presently described method and/or device and/orassembly can be directed to a vehicle, a car and/or a self-driving car.That is, the mapping method can be used by a self-driving car to buildmap data, and/or to navigate and/or to perform SLAM.

However, in other embodiments, the device and/or the assembly describedherein, that is, the mobile robot and/or the assembly of mobile robotsare substantially different from a car and/or a self-driving car. Thatis, in such embodiments, the mobile robot is significantly smaller thana car. In such embodiments, typical dimensions of the robot may be asfollows. Width: 20 to 100 cm, preferably 40 to 70 cm, such as about 55cm. Height: 20 to 100 cm, preferably 40 to 70 cm, such as about 60 cm.Length: 30 to 120 cm, preferably 50 to 80 cm, such as about 65 cm. Insuch embodiments, the mobile robot is also sufficiently lighter than acar and/or a self-driving car. In such embodiments, the weight of therobot may be in the range of 2 to 50 kg, preferably in 5 to 40 kg, morepreferably 7 to 25 kg, such as 10 to 20 kg. In such embodiments, therobot can also be adapted to operate on the sidewalks unlike a carand/or a self-driving car. It can further have a velocity of no morethan 20 km/h, such as no more than 10 km/h, preferably between 4 and 6km/h.

That is, embodiments where the present invention can be applied to carsand/or self-driving cars are substantially different from embodimentswhere it can be applied to smaller, and/or lighter, and/or slower mobilerobots.

It is known in the art that self-driving cars can use a plurality ofsensors to autonomously navigate on public roads. Often, self-drivingcars use Lidar sensors as a primary or one of the primary means oflocalization. The current invention presents a cost, space and equipmentefficient way to localize a mobile robot with a precision of a fewcentimetres. The current invention can be applied to self-driving cars,however, the technology presently used for self-driving cars can becumbersome and impractical to use on a mobile delivery robot operatingon sidewalks.

The above features along with additional details of the invention, aredescribed further in the examples below, which are intended to furtherillustrate the invention but are not intended to limit its scope in anyway.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a perspective view onto a robot embodiment in accordancewith the present invention;

FIG. 2 shows an embodiment of different camera viewing angles;

FIG. 3 shows an embodiment of straight lines extracted from an exemplaryimage using the described invention;

FIG. 4 shows a schematic description of an embodiment of a mappingmethod;

FIG. 5 shows a schematic description of an embodiment of a localizationmethod;

FIG. 6 shows an embodiment of a localization method according to theinvention;

FIG. 7 shows an embodiment of the error capping function according toone embodiment of a mapping method.

DESCRIPTION OF VARIOUS EMBODIMENTS

In the following, exemplary embodiments of the invention will bedescribed, referring to the figures. These examples are provided toprovide further understanding of the invention, without limiting itsscope.

In the following description, a series of features and/or steps aredescribed. The skilled person will appreciate that unless required bythe context, the order of features and steps is not critical for theresulting configuration and its effect. Further, it will be apparent tothe skilled person that irrespective of the order of features and steps,the presence or absence of time delay between steps, can be presentbetween some or all of the described steps.

FIG. 1 shows an embodiment of the robot 100 according to the invention.The robot comprises wheels 1 adapted for land-based motion. Frame 2 canbe mounted on the wheels 1. Body 3 can be mounted on the frame 2. Aprocessing component 15, a memory component 16 and a communicationcomponent 17 can be mounted on the frame 2 and/or on the body 3 of therobot 100. The memory component 16 stores data and instructions,including instructions implementing an iterative algorithm 16 a thatresides in the memory component. Body 3 can comprise an enclosed space(not shown) adapted to transport a delivery. Lights 4 can be placedaround body 3 and/or frame 2. Lights 4 can for example be LED lights andcan illuminate the environment in which the robot finds itself. This canbe useful to indicate the presence of the robot in the dark and/orassist visual localization through better illumination. A plurality ofcameras can be placed around body 3. In this embodiment, 9 (nine)cameras are present.

A first camera 10 can be positioned near the front of the robot on thebody 3. The first camera can provide an approximately horizontal viewaway from the robot. A second camera 20 and a third camera 30 arepositioned on the two sides of the first camera 10 similarly near thefront of the robot.

Second camera 20 and third camera 30 can be angled 10-50° downwards,preferably 20-40° downwards with respect to the first camera's 10orientation, i.e. they can be angled downwards with respect to ahorizontal view. Second camera 20 and third camera 30 can be stereocameras. They can be separated by a distance of 5-10 cm. The stereocameras facilitate triangulation of objects by comparing the featurespresent on the visual images from the stereo cameras.

A fourth camera 40 and a fifth camera 50 are placed on the left side ofthe robot's body 3 with respect to a forward direction of motion. Thefourth camera 40 and the fifth camera 50 can also be stereo cameras.They can be separated by a distance of 15-20 cm.

On the right side of the robot's body with respect to the direction ofmotion, a sixth camera (not shown) and a seventh camera (not shown) areplaced in a position that is complementary to positions of cameras 40and 50. The sixth camera and the seventh camera can also be stereocameras preferably separated by a distance of 15-20 cm.

On the back of the robot, an eighth camera (not shown) and a ninthcamera 90 can be placed. The eighth camera and the ninth camera 90 canalso be stereo cameras preferably separated by a distance of 5-10 cm.One or more cameras can be arranged in a portrait orientation. Thismeans that the vertical viewing angle can be larger than the horizontalone. In the shown embodiment, the further through seventh side camerascan be placed in a portrait orientation. The other cameras (firstthrough third and eighth and ninth) can be placed in a landscapeorientation. This means that the horizontal viewing angle can be largerthan the vertical one.

FIG. 2 shows an embodiment of the robot according to the invention. FIG.2 demonstrates viewing angles of a camera setup as shown in FIG. 1 . Allof the cameras' viewing angles are shown. The viewing angles can be inthe range of 40-80° by 60-100°, preferably about 60° by 80°. The viewingangle 11 corresponds to the first camera 10. The viewing angles 21 and31 correspond to the cameras 20 and 30 respectively. Those two camerascan be arranged in a stereo manner, which is why FIG. 2 demonstrates theviewing angles intersecting. A similar arrangement can be achieved withthe eighth and ninth cameras—these can also be stereo cameras placedtowards the back of the robot on its body 3. Therefore, viewing angles81 and 91 corresponding to the eighth and ninth cameras, respectively,are also shown as intersecting. The two pairs of side cameras—one pairbeing the fourth and fifth cameras 40, 50 and the second pair being thesixth and seventh cameras (not shown) can be placed in a stereo positionin a portrait orientation. Their corresponding viewing angles 41 and 51,and 61 and 71 respectively similarly intersect. The robot has camerasmounted thereon which point in different directions and whose viewingangles do not intersect, i.e., their fields of view are non-overlapping.For example, cameras 20 and 90 point in opposite directions (one forwardand one rearward) and have non-overlapping fields of view.

FIG. 3 shows an embodiment of straight lines 100 that can be extractedduring the operation of the robot. Straight lines 100 can belong topermanent objects (such as houses, fences, sidewalks) and/or transitoryobjects (such as cars, shadows). The invention is adapted to becalibrated using multiple test cases of the images—improving itsaccuracy in detecting the lines and identifying the lines belonging topermanent objects. The extracted straight lines thus correspond tocontours of permanent (non-transitory) physical objects in the images.

FIG. 4 shows an embodiment of a mapping method according to theinvention. The first step S1 comprises taking visual images with thecameras placed on the robot. The visual images can be takensimultaneously. In a preferred embodiment, the robot comprises 9 (nine)cameras taking simultaneous images. The second step S2 comprisescombining the visual images into one file for preprocessing. This stepcan be done to speed up the process. After the preprocessing, thecombined file can be separated into the individual images again. Thethird step S3 comprises extracting lines from the individual images.This step can be done using first an edge detecting algorithm such asfor example the Canny algorithm and then using a line extractingalgorithm on the result of the edge detecting algorithm. The lineextracting algorithm can for example be the Hough transform. The fourthstep S4 comprises combining the extracted lines to build map data of thearea the visual images were taken in.

The precise positions of the cameras on the robot and with respect toeach other can be known, which enables combining the extracted lines ina coherent manner in one coordinate system. This coordinate system canbe arbitrary, as long as it is consistent and can be converted into astandard system such as GPS coordinates. The method comprising steps S1,S2, S3, and S4 can be repeated every time a new set of visual images istaken by the cameras. In a preferred embodiment, this is repeated 1-10times per second. The robot can thus build a consistent map data of itsarea of operation. If multiple robots are operating in one area ofoperation, they can exchange map data and update it when changes aredetected. The robots can thus benefit from the map data taken by otherrobots. Map data of different operating areas can be combined intoglobal map data comprising all of the operating areas of the robots.

FIG. 5 shows an embodiment of a localization method according to theinvention. Steps S1, S2, and S3 can be the same as in the mapping methodof FIG. 4 . The localization method can be used when the robot comprisesmap data stored within its memory component. The fifth step S5 comprisescomparing the straight lines extracted from the visual images (e.g., theaforementioned solid lines 120) to map data stored within the robot'smemory component (e.g., the dotted lines 110). The map data storedwithin the memory component corresponds to different pose possibilitiesof the robot. The robot can then use a particle filter algorithm toevaluate the likelihood of each pose being the true one. In the sixthstep S6 the most likely pose is picked based on the probabilisticanalysis of known pose possibilities. This most likely pose will providelocalization of the robot at the time the images are taken. Thelocalization is rapid, and is typically complete within a very shorttimeframe, or at least before the next sets of images are processed(which can occur every 0.1 to 1 second).

If, for some reason, the robot is transiently unable to performimage-based localization, for example if the robot is unable to accessor download a map to memory for performing localization during transit,the robot can navigate using other means of localizing that are alsoimplemented on the robot (e.g., one or more of GPS coordinates,accelerometer data, gyroscope data, odometer data, time of flight cameradata, magnetometer data and/or at Lidar data. Once the robot is able toresume image-based localization, its course can be readjusted ifnecessary, based on the more accurate localization data, taking intoaccount its intended route of navigation.

FIG. 6 shows an embodiment of a localization method according to theinvention. Steps S1, S2, and S3 can be the same as in the mapping methodof FIG. 4 and in the localization method of FIG. 5 . The localizationmethod can be used when the robot comprises map data stored within itsmemory component.

The seventh step S7 can comprise receiving location related data fromone or more dead reckoning components. Those can comprise at least oneodometer, at least one accelerometer, and/or at least one gyroscope. Theeighth step S8 can comprise combining location related data obtainedfrom the lines extracted from the visual images and location relateddata received from the one or more dead reckoning components weightedbased on the errors associated with each of them. The ninth step S9 cancomprise forming a hypothesis on the robot's pose based on the combineddata. The last two steps can be performed using for example a particlefilter algorithm as described above and below.

In one embodiment, the robot can receive location data each time stepfrom the dead reckoning component. This location data can comprise anerror estimate associated with it. Optimal time step duration can bedetermined by calibration. In a preferred embodiment, a time step cancomprise 0.01-0.1 seconds, more preferably 0.01-0.05 seconds. Thelocation data can be taken as a starting point for robot pose estimationat each time step. The dead reckoning component can comprise at leastone odometer and/or at least one gyroscope. The dead reckoning componentcan then be a control sensor as described in the particle filterdescription.

The robot can further take visual images using at least two cameras. Therobot's processing component can then extract features from the visualimages. In a preferred embodiment, straight lines are extracted from thevisual images and comprise location related data. The lines seen on agiven image and/or a given combination of images can be compared withthe lines that should be seen (based on the map) based on the givenparticle's pose. Quantitatively this can be represented as a probabilityof seeing the particular lines given the particle pose. This probabilitycan be calculated approximately by a fitness function. It can be appliedto the particle weights as described before. Normalization can be doneto reduce correlations within a camera frame—one camera receiving manylines (like for example from a picket fence) should not dominate overanother camera input that received only a few lines (that for exampleonly saw a couple of building corners). This is furthermore done to keepthe error estimate within a reasonable range (for numerical stability).In one embodiment, the fitness function does approximately thefollowing: associating a line from a camera image with a line on themap, calculating the error between the two, summing up all the errors(for example using the square summed method), normalizing the sumsacross all of the images taken at a point in time, adding them up, andfinally taking an exponential of the negative sum.

The processing component can then combine the data from the deadreckoning component and from the line based localization along withtheir respective errors to obtain an estimation of the possible robotposes. This can be done using the particle filter method. During thisstep, input from further sensors and/or components can be considered.For example, the robot can consider the location or pose related datayielded by a GPS component, a magnetometer, a time of flight camera,and/or an accelerometer.

At each time step, the robot can update the weight of all the particleswithin the particle filter and ends up with a distribution of likelyrobot poses. A resampling step can be done when a certain criterion isreached to make sure that the particle filter does not fail.

FIG. 7 shows an embodiment of the error capping function according toone embodiment of a mapping method. As discussed above, the robot cancomprise various sensors including the cameras, and, potentiallyincluding at least one but not limited to the GPS module, the odometrysensors, the accelerometers, the magnetometers, the ultrasonic sensors,the Time of Flight cameras, and/or the altitude sensors. Those sensorsyield measurements with an associated error. Some of these errors can besystematic, as described above, resulting in values far outside thenormal range of the sensors. This can lead to problems with building themap, as the map may end up distorted, so it cannot be used fornavigation, and/or it may end up with too few landmarks and/or theiterative algorithm and/or algorithms can fail. Additionally oralternatively, when associated lines extracted from camera images, theassociations can fail in such a way that the value is also outside thereasonable range. Therefore, an error capping mechanism can be used toavoid failure of the algorithm due to too large errors. In oneembodiment, an error capping function α(x) can be introduced. The leftpart of FIG. 7 shows one possible embodiment of the error function 300,the arctangent. The variable of the function, that is the error, x isdepicted by the line 200. Around the origin, the variable 200 and theerror capping function 300 are very close to each other, that is, theypractically coincide. However, after the variable x 200 exceeds acertain value x_(L), it continues growing linearly, while the errorcapping function 300 converges to a constant value. This constant valueis denoted as ϵ in the figure. The range of linearity x_(L) and themaximum error value ϵ can be adjusted as needed for a sensible cappingof errors. Note, that the arctangent is only one possible function to beused as the error capping function 300. Other functions that behave in asimilar way can be used. That is, other functions that approximatelycoincide with the variable around the origin and converge to a certainfinite value after some time can be used. Preferably, the error cappingfunction 300 is differentiable. Preferably, its first derivative iscontinuous. Preferably, the function 300 is strictly monotonic, that is,increases with increasing variable 200.

The right part of FIG. 7 shows the square of the variable x 210 and thesquare of the error capping function α(x) 310. Around the origin, thetwo still almost coincide, and as the parabola x² 210 diverges, thesquare of the error capping function (in this case the square of thearctangent) α(x)² converges to ϵ².

As used herein, including in the claims, singular forms of terms are tobe construed as also including the plural form and vice versa, unlessthe context indicates otherwise. Thus, it should be noted that as usedherein, the singular forms “a,” “an,” and “the” include pluralreferences unless the context clearly dictates otherwise.

Throughout the description and claims, the terms “comprise”,“including”, “having”, and “contain” and their variations should beunderstood as meaning “including but not limited to”, and are notintended to exclude other components.

The term “at least one” should be understood as meaning “one or more”,and therefore includes both embodiments that include one or multiplecomponents. Furthermore, dependent claims that refer to independentclaims that describe features with “at least one” have the same meaning,both when the feature is referred to as “the” and “the at least one”.

It will be appreciated that variations to the foregoing embodiments ofthe invention can be made while still falling within the scope of theinvention can be made while still falling within scope of the invention.Features disclosed in the specification, unless stated otherwise, can bereplaced by alternative features serving the same, equivalent or similarpurpose. Thus, unless stated otherwise, each feature disclosedrepresents one example of a generic series of equivalent or similarfeatures.

Use of exemplary language, such as “for instance”, “such as”, “forexample” and the like, is merely intended to better illustrate theinvention and does not indicate a limitation on the scope of theinvention unless so claimed. Any steps described in the specificationmay be performed in any order or simultaneously, unless the contextclearly indicates otherwise.

All of the features and/or steps disclosed in the specification can becombined in any combination, except for combinations where at least someof the features and/or steps are mutually exclusive. In particular,preferred features of the invention are applicable to all aspects of theinvention and may be used in any combination.

What is claimed is:
 1. A mapping method performed by a processor on amobile robot for generating a map of an area in which the mobile robottravels, comprising: (a) extracting straight lines from visual imageswith at least one processing component; (b) determining which of theextracted straight lines are associated with transitory physical objectsand discarding those lines associated with transitory physical objects,and detecting permanent physical objects in the visual images based atleast in part on the remaining extracted straight lines; (c) receivinglocation-related data from one or more sensors adapted to measureparameters for building map data, said one or more sensors being fromthe group consisting of a GPS component, an accelerometer, a gyroscope,an odometer, a magnetometer, a pressure sensor, an ultrasonic sensor, atime of flight camera sensor, and a Lidar sensor; (d) combining thelocation-related data with data from the visual images; (e) generatingmap data based on the detected permanent physical objects and receivedlocation-related data; and (f) sending the generated map data and/orimage data to an external server configured to refine existing map datausing the generated map data and/or image data.
 2. The mapping methodaccording to claim 1, wherein step (b) comprises: detecting that acertain number of extracted lines all belong to the same physical objectand merging them into one line.
 3. The mapping method according to claim2, comprising using an iterative algorithm to associate the extractedstraight lines with physical objects.
 4. The mapping method according toclaim 3, wherein the iterative algorithm optimizes positions of thephysical objects based on the extracted straight lines.
 5. The mappingmethod according to claim 3, wherein the iterative algorithm comprises:combining lines belonging to the same physical object from images takenby different cameras; and discarding lines belonging to transientobjects and/or light or camera effects.
 6. The mapping method accordingto claim 1, comprising: capturing the visual images by at least twocameras; and combining the visual images in a single reference framebased on the relative placement of the at least two cameras.
 7. Themapping method according to claim 1, wherein the method performssimultaneous mapping and localization (SLAM).
 8. The mapping methodaccording to claim 1, wherein the method further comprises receiving thevisual images of the area.
 9. The mapping method according to claim 1,wherein the method comprises: providing a mobile robot configured tonavigate outdoors on a sidewalk to deliver an item to a predetermineddelivery location, the mobile robot having a body and a space forholding the item while in transit, the mobile robot further comprisingat least two cameras mounted on the body and said at least oneprocessing component; operating the mobile robot in the area, whereinthe area is an outdoor operating area; and taking the visual images withthe at least two cameras.
 10. A mobile robot comprising: at least twocameras adapted to take visual images of an operating area; at least oneprocessing component adapted to: extract straight lines from the visualimages taken by the at least two cameras; determine which of theextracted straight lines are associated with transitory physical objectsand discard those lines associated with transitory physical objects;detect permanent physical objects in the visual images based at least inpart on the remaining extracted straight lines; and generate map datacorresponding to the detected permanent physical objects; acommunication component adapted to send the generated map data and/orimage data to an external server which is configured to refine existingmap data using the generated map data and/or image data sent by therobot's communication component; and one or more sensors adapted tomeasure parameters for building map data, said one or more sensors beingfrom the group consisting of a GPS component, an accelerometer, agyroscope, an odometer, a magnetometer, a pressure sensor, an ultrasonicsensor, a time of flight camera sensor, and a Lidar sensor.
 11. Themobile robot according to claim 10, wherein the processing component isadapted to: determine whether an extracted line belongs to a permanentphysical object or to a transitory physical object; and upon detectingthat a certain number of extracted lines all belong to the samepermanent physical object, merge them into one line.
 12. The mobilerobot according to claim 10, wherein the map data comprises: one or moreof vectors, point features and grid features associated with permanentphysical objects, and defined with respect to a coordinate system,and/or error estimates for one or more of vectors, point features andgrid features associated with said permanent physical objects.
 13. Themobile robot according to claim 10, wherein the map data furthercomprises visibility information related to locations from whichpermanent physical objects can and/or cannot be seen.
 14. The mobilerobot according to claim 10, wherein the at least one processingcomponent is adapted to generate said map data, while the mobile robotis in transit to a predetermined delivery location.
 15. The mobile robotaccording to claim 10, wherein the at least one processing component isadapted to generate said map data using an iterative algorithm.
 16. Themobile robot according to claim 10, configured to navigate using therefined map data received from the external server.
 17. The mobile robotaccording to claim 10, wherein the robot is autonomous and/orsemi-autonomous.
 18. The mobile robot according to claim 10, configuredto navigate outdoors on a sidewalk and deliver an item to apredetermined delivery location, the mobile robot having a body and anenclosed space for holding the item while in transit.
 19. The mobilerobot according to claim 10, wherein: the robot is adapted to travelwith a speed of no more than 10 km/h; the robot has at least 4 pairs ofstereo cameras, members of each pair of stereo cameras located on themobile robot so as have overlapping fields of view and provide depthinformation; the at least 4 pair of stereo cameras includes a first pairof stereo cameras mounted on a front of the body, second and third pairsof stereo cameras mounted on opposite sides of the body and a fourthpair of stereo cameras mounted on a back of the body; and each camera isadapted to capture 3 to 5 images per second.